Head and Hands 3D Tracking in Real Time by the EM algorithm

This article present a method for real time hand and head tracking, in three dimensions, using two cameras. This tracking is intended as a first step for a gesture recognition system, using the trajectories of the hands, or as input to a real time clone animation system. The method used is based on simple preprocessings followed by the use of a statistical model linking the observations (obtained by the preprocessing stage) to the parameters: the position of the hands and the head. Preprocesssing consist of background substraction followed by skin color detection, using a simple color lookup table. The statistical model is composed of three ellipsoids, one for each hand and one for the head. Each one is projected on each camera plane as an ellipse. A gaussian probability density with the same center and size is associated with each ellipse. The parameters of the model (the position and orientation of the ellipsoids) are adapted to the pixel detected by the preprocessing stage. This adaptation takes into account simultaneously the detected pixels for the two cameras, and is based on the maximum likelihood principle. The EM algorithm is used to obtain the parameters corresponding to the maximum of the likelihood. The hands and head tracking is realised in near real time (12 Hz) on a single workstation. The method used is quite generic and can be extended using other informations such as movement, or by using more complex statistical models such a gaussian skin color model of each ellipsoid. A body model, linking the position and angles of the articulations of the body to the observed detected pixels could also be used for better tracking.